Previous Page

Mulligan Security - 2025-02-07

Server Monitoring

What is server monitoring?

When deploying compute resources (bare-metal, VPSes or more abstract work units) you will have to manage a living system. This system will always have the following characteristics:

• Limited ressources: the amounts of RAM and CPU cycles, network bandwidth as well as storage space are neither infinite nor free.
• Evolving requirements: depending on how you use your services, how many concurrent users you have you might need more or less ressources than what you initially purchased
• Nominal operating parameters: range of RAM and CPU use, temperatures and so on in which your service performs as expected

The first item is fixed and only linked to your financial constraints. The other two are constantly evolving and thus must be monitored.

What if I don't?

If you don't properly monitor your infrastructure you will face the following consequences sooner or later:

• service instability: you won't notice when things start going awry
• costs overrun: you will end up paying more than you need to in order to deliver the same service
• undetected attacks: attacks that impact your services can go unnoticed when the cues (eg: high RAM consumption from a cryptojacking) are not picked up

How do I do it?

How you monitor your systems can vary based on your technical requirements. It can be as simple as logging in once a week, check the output of some diagnostic command and calling it a day.
This will give you a snapshot but you will miss a lot of important information.

You can also set up a complicated system that reports current metrics, trends and gives you capacity planning alerts based on the data obtained! You will have to find the middle-ground yourself, this article will propose one that you can tweak whichever way you need.

Risks of doing it improperly

Accessing your server for monitoring purposes is, from a risk perspective, pretty much the same as doing any other administration task or interacting with the services hosted therein. If done improperly (say logging in over the clearweb from your home address) you've just given anyone looking an undeniable link between your overt identity and your clandestine activities.

A fail-closed system is what you should strive for: opsec best practices should be the default and if there's a technical issue preventing you from following them (attack on tor, flaky network, client or server-side misconfiguration) the system should prevent access at all in order to keep you safe.

Basic tools

Let's look at a cryptojacked server... In this case the intruder did not take any precautions or try to hide their activity. This often happens with basic scripts that scan the internet in large-scale low-cost credential stuffing attacks.

glances

Here we will look at glances. Glances is a python tool that gives nice looking visuals with information about server status

Pros

• looks nice

Cons

• Requires python
• not part of the POSIX convention
• somewhat resource intensive for limited hardware
• Only shows point-in-time data, no history
• No alerting whatsoever

top

Now, an oldie but a goodie: top! Wherever you find a unix you'll find top, from MacOS to BSD...

Pros

• Lightweight
• POSIX Compliant

Cons

• Ugly
• limited ordering/filtering features compared to glances
• Only shows point-in-time data, no history
• No alerting whatsoever

Risks

Whenever you connect to your server, such as for monitoring or other administrative tasks, if you do so through the clearweb then you are liable to being recorded. Even when using SSH you will leave a trail of metadata all the way back to your access point. That might be enough to get your door busted down the line.

In the following part of the post we will look into:

• How to connect to your server safely and anonymously (see this article)
• How to set up advanced monitoring tools so you don't have to keep an eye on a bunch of tmux sessions with glances/top open